Approximate Detection of Machine - morphed Variants of Malicious Programs

A morphing malware is malicious software that uses a code morphing program, ormorphing engine, to transform its own code into a morphed variant. The goal ofthis transformation is to evade recognition by malware detectors. This dissertationproposes and evaluates a new method for detecting morphed malware variants. The method uses information about the morphing engine to recognize variants createdby that engine. In particular, it is shown that implementation of the requirementsof good design practices of morphing malware can be capitalized upon to efficientlydiscriminate programs generated by a morphing engine implementing theserequirements from programs that have not been generated by the engine. Exactrecognition techniques implementing this method are proposed and shown to becomputationally costly. Approximate efficient variations on these techniques arethen proposed and successfully evaluated to recognize programs generated by a realworld morphing engine, W32.Evol. Finally, the variation of a malware’s instructiondistribution underlying a probabilistic morphing engine is modeled as a Markovchain. Techniques from Markov chain theory are suggested to enable the use, fordetection purposes, of the distribution of the instruction-frequency vectors of the various generations of variants of morphed malware generated by a probabilisticmorphing engine.